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Abstract 


when  a  new  computer  software  package  is  developed,  a  testing  procedure 
is  often  put  into  effect  to  eliminate  the  faults,  or  bugs,  in  the  package. 
One  common  procedure  is  to  try  the  package  on  a  set  of  well  known  problems 
to  try  to  see  if  any  errors  result.  This  goes  on  for  some  fixed  time  with 
all  detected  errors  being  noted.  Then  the  testing  stops  and  the  package 
is  carefully  checked  to  determine  the  specific  bugs  that  were  responsible 
for  the  observed  errors,  and  the  package  is  then  altered  to  remove  these 
bugs.  A  problem  of  great  importance  is  the  estimation  of  the  error  rate 
of  this  revised  software  package. 

To  model  the  above,  we  suppose  that  initially  the  package  contains  m  , 
an  unknown  number,  of  bugs  which  cause  errors  to  occur  in  accordance  with 
independent  Poisson  process  having  unknown  rates  ,  i  =  1,  ...,  m  .  We 

suppose  that  the  package  is  to  be  run  for  t  time  units  and  that  each  error 
is,  independently,  detected  with  some  known  probability  p  .  At  the  end  of 
this  time,  a  careful  check  of  the  package  is  made  to  determine  the  specific 
bugs  that  caused  the  detected  errors  (that  is,  a  debugging  takes  place). 
These  bugs  are  then  removed  and  the  problem  of  interest  is  to  determine  the 
error  rate  for  the  revised  package.  In  this  paper  we  show  how  to  estimate 
this  quantity  under  a  variety  of  assumptions  as  to  what  is  learned  when  the 
debugging  occur. 
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STATISTICAL  ESTIMATION  OF  SOFTWARE  RELIABILITY 


by 

Sheldon  M.  Ross 

University  of  California,  Berkeley 

0.  INTRODUCTION 

When  a  new  computer  software  package  is  developed,  a  testing  procedure 
is  often  put  into  effect  to  eliminate  the  faults,  or  bugs,  in  the  package. 
One  common  procedure  is  to  try  the  package  on  a  set  of  well  known  problems 
to  try  to  see  if  any  errors  result.  This  goes  on  for  some  fixed  time  with 
all  detected  errors  being  noted.  Then  the  testing  stops  and  the  package  is 
carefully  checked  to  determine  the  specific  bugs  that  were  responsible  for 
the  observed  errors,  and  the  package  is  then  altered  to  remove  these  bugs. 
However,  as  we  cannot  be  certain  that  all  the  bugs  in  the  package  have  been 
eliminated,  a  problem  of  great  importance  is  the  estimation  of  the  error 
rate  of  the  revised  software  package. 

To  model  the  above,  let  us  suppose  that  initially  the  package  contains 
m  ,  an  unknown  number,  of  bugs  which  we  will  refer  to  as  bug  l,bug  2,  ..., 
bug  m  .  Suppose  also  that  bug  i  will  cause  errors  to  occur  in  accordance 
with  a  Poisson  process  having  an  unknown  rate  A  ,  i  ■  1,  ...,  m  .  Then, 
for  instance,  the  number  of  errors  due  to  bug  i  that  occur  in  any  s 
units  of  operating  time  is  Poisson  distributed  with  mean  A^s  .  Also  sup¬ 
pose  that  these  Poisson  processes  caused  by  bugs  i  ,  i  =  1,  ...,  m  are 
independent.  Also  we  suppose  that  the  package  is  to  be  run  for  t  time 
units  and  we  suppose  that  each  error  is,  independently,  detected  with  some 
known  probability  p  .  At  the  end  of  this  time,  a  careful  check  of  the 
package  is  made  to  determine  the  specific  bugs  that  caused  the  detected 


errors  (that  is,  a  debugging  takes  place).  These  bugs  are  then  removed  and 
the  problem  of  interest  is  to  determine  the  error  rate  for  the  revised  pack- 


The  above  problem  is  considered  in  Section  1  and  a  preliminary  estima¬ 
tion  is  presented.  In  Section  2,  we  make  the  added  assumption  that  once  a 
given  bug  has  been  found,  its  error  rate  becomes  known.  Under  this  assump¬ 
tion,  we  show  how  to  improve  upon  the  estimator  of  Section  1.  We  also 
present,  in  Section  2,  an  estimator  different  than  that  in  Section  1  which 
can  be  used  when  error  rates  are  not  learned  and  one  that  can  be  used  when 
debuggings  necessarily  occur  whenever  an  error  is  detected.  In  Section  3, 
we  consider  the  situation  where  a  debugging  occurs  whenever  an  error  is 
detected  but  it  need  not  be  successful.  In  Section  4,  we  start  with  a 
Bayesian  model  which  initially  assumes  that  the  number  of  errors  is  Poisson 
distributed  with  known  mean  c  ,  and  given  the  number  of  bugs  the  failure 
rates  of  the  bugs  are  independent  with  a  common  known  distribution  G  .  We 
then  successively  allow,  in  Section  4.1,  the  Poisson  parameter  c  to  be 
unknown,  and,  in  Section  4.2,  both  c  and  G  to  be  unknown.  In  both 
these  latter  cases,  we  suppose  that  a  bugs  failure  rate  becomes  known  when 
the  bug  is  detected.  Interestingly,  our  estimate  when  both  c  and  G  are 
unknown  is  identical  to  the  one  given  in  Section  2.  In  Section  5,  we  show 
how  the  data  at  time  t  can  be  used  to  estimate  what  the  total  error  rate 
would  be  at  time  t  +  s  if  the  testing  were  to  continue  for  an  additional 
time  s  and  also  present  an  estimator  for  the  expected  number  of  new  bugs 
that  are  discovered  in  (t  ,  t  +  s)  . 

For  a  survey  of  other  statistical  procedures  used  in  software  reliabil¬ 
ity  estimation,  the  interested  reader  should  see  Ramamoorthy  and  Bastani  [7], 


• 
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1.  A  PRELIMINARY  MODEL 


Let 


^(t) 


(1  if  bug  i  has  not  caused  a  detected  error  by  t 


0  otherwise  . 


The  quantity  we  wish  to  estimate  is  thus 


A(t)  =  l  A.iK  (t) 
i 


the  error  rate  of  the  final  package.  To  start  note  that 


E[A(t)]  =  l  A^t^Ct)] 


(1) 


r  ‘  ipt 

=  I  V  • 

i 


Now  each  of  the  bugs  that  are  discovered  would  have  been  responsible  for  a 


certain  number  of  detected  errors.  Let  us  denote  bv  M^(t)  the  number  of 


bugs  that  were  responsible  for  j  detected  errors,  j  >_  1  .  That  is,  M^(t) 


is  the  number  of  bugs  that  caused  exactly  1  detected  error,  M£(t)  is  the 


number  that  caused  2  errors,  and  so  on,  with  £  jM  (t)  equalling  the  total 


number  of  detected  errors.  To  compute  E[M1(t)]  ,  let  us  define  the  indi¬ 


cator  variables,  I  (t)  ,  i  _>  1  ,  by 


I±(t) 


|l  bug  i  causes  exactly  1  detected  error 
|o  otherwise  . 


a* 

c.v 


*  ' 
‘  V 


4 


Then, 


M1(t) 


I 


and  so 


(2) 


-A.pt 

E[M  (t)]  =  l  E[I  (t) ]  =  l  A  pte 
i  i 


Thus,  from  (1)  and  (2)  we  obtain  the  intriguing  result  that 


A(t)  - 


Mx(t) 


Pt 


=  0 


M1(t) 

This  suggests  the  possible  use  of  — — —  as  an  estimate  of  A(t)  .  To 


Pt 


M1(t) 

determine  whether  or  not  — — —  constitutes  a  "good"  estimate  of  A(t)  we 


Pt 


shall  look  at  how  far  apart  these  2  quantities  tend  to  be.  That  is,  we  will 
compute 


(A<t)  -  — )  =  var  (A(t)  -  -V) 


Var  (A ( t) )  -  Cov  (A(t),M1(t))  +  — Var  (M]_(t)) 


P  t 


Now 


v  2  r  2  /  A-i  CP\ 

Var  (A(t))  =  l  Var  (t ) )  =  £  A^  /l  -  e  J 


r  v  'Xlpt/  "Xipt\ 

Var  (M^t))  =  l  Var  (I^t))  -  l  A^te  fl  -  A^te  I 


V.'.] 
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Cov  (A(t),M1(t)) 


=  Cov 


(l  yt>) 


=  II  Cov  (X  ij>  (t),I  (t)) 
i  j  J 

=  I  X  Cov  (t )  ,  I .  (t) ) 
i 

-X.pt  -X.pt 

=  -  I  V 

i 


where  the  last  two  equalities  follow  since  ij^(t)  and  I^Ct)  are  inde¬ 
pendent  when  i  ^  j  as  they  refer  to  different  Poisson  processes,  and 
4>i(t)Ii(t)  =  0  .  Hence  we  obtain  that 


I  m  <tn 

(Mt)  -  V) 


r  ,2  -1itI>  1  r  ,  "AitP 

’  i  V  +  ^  l  X1S 


E[Mx(t)  +  2M2(t>] 

2~2 
P  t 


where  the  last  equality  follows  since 

-X.tp  7 

E[M  (t) ]  =  l  fe  1  (X ,tp)  /2  . 
i 

Thus  we  can  estimate  the  average  square  of  the  difference  between 

Ml(t)  2  2 

A(t)  and  — — —  by  the  observed  value  of  M^Ct)  +  2M2(t)  divided  by  p  t 

Remark: 

The  above  analysis  is  similar  in  spirit  to  that  done  in  Robbins  [4]. 
Very  similar  results  have  also  been  presented  by  Diaconis  in  a  set  of 
unpublished  notes  on  decision  theory. 
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2.  ERROR  RATES  LEARNED  UPON  DEBUGGING 

Let  us  now  suppose  that  the  failure  rate  due  to  a  bug  becomes  known 
once  the  bug  has  been  discovered.  That  is,  we  are  supposing  that  based 
on  our  experience  we  are  able  to  accurately  estimate  the  failure  rate  due 
to  any  particular  bug  once  it  has  been  discovered. 

Let  R  denote  the  number  of  bugs  discovered  by  time  t  and  let 
A^,  ....  AR  be  their  failure  rates.  Then 

-X  pt 

X  pte 

E[M1(t)  |  R,A1,  ...,  Ar]  =  l  - (1  -  *±(t))  . 

1  1  -  e  1 

We  shall  consider  —  E[M..(t)  I  R.A.,  ....  An]  as  an  estimate  of  A(t)  . 

pt  l  ±  K 

Since 

E[E[M1(t)  |  R,Ar  ...,  Ar]  ]  =  ElMjU)]  =  ptE[A(t)] 


its  square  error  loss  is  as  follows: 


> 


7 


It  should  be  noted  that  as 


1  -  e 


— <  A2  +  —  ,  A  >  0  ,  t  >  0 

-Apt  -  pt  —  - 


it  follows  that 


-A .  pt 
1 


A.e  R  A . e 

I  -X.pt  -  »i(t»  -  l 


-A^t 


l  i  i 

1  -  e 


^  —A  Dt 

i=l  , 

1  -  e 


is  a  better  estimator  of  A(t)  than  is  M^(t)/tp  ,  and  should  thus  be  pre¬ 
ferred  whenever  the  finding  of  a  bug  also  reveals  its  failure  rate. 


Remarks : 


The  above  estimator  can  be  used  whether  debugging  is  performed  when¬ 
ever  an  error  is  detected  and  the  bug  removed  or  if  the  debugging  is 
performed  at  time  t  .  In  this  latter  case,  another  estimator  is 
obtained  by  first  noting  that,  independent  of  the  bugs  causing  the 

errors,  detected  errors  will  occur  at  a  Poisson  rate  p  £  A.  . 

i 

Hence,  letting  D(t)  =  £  iM  (t)  denote  the  number  of  detected  errors 

.  1  R 

D  ( t )  r  r 

by  t  ,  then  — —  can  be  used  to  estimate  )  A.  .  As  )  A.  is 
pt  h  i  l 

l  i=l 

the  error  rate  due  to  those  bugs  that  have  caused  detected  errors. 


it  thus  follows  that 


-  I  A,  is  an  estimator  of  A ( t )  .  To 
i=l  1 


evaluate  its  worth,  note  that 


R  1  /  -A . pt \ 

E  l  A  =  E  l  A  (1  -  *  (t))  =  l  XI  -  e  1 

i=l  J  Li  J  i  ' 
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and  so 


.  R  -Ltp 

s  ~~  -  l  Ai  =  l  V  1  =  EtA(t)  ] 

F  i=l 


(m .  ji  Ai .  kM) .  var  (mu)  .  i. 


However, 


E[D(t)  1  Eiyt>  +  ^(t)1 

i  tp  '  p2t2  -  p2t2 


H  -  — ) 


=  Var 


it  follows  that  this  estimator  is  not  even  as  good  as 


M1(t) 


Of  course,  the  estimator  could  be  improved  somewhat  by  considering 


If  we  are  not  willing  to  suppose  that  we  can  accurately  estimate 
the  failure  rate  of  a  discovered  bug,  another  approach  is  to  sup¬ 
pose  that  you  can  express  your  feelings  in  terms  of  a  probability 
iistribution  on  A^  .  This  suggests  the  estimator 


-A .  tp 

R  A .  e  1 

i-i  \  ~  IVp 

1  -  e 


where  is  ones  feelings  about  the  ith  discovered  bug  (after  it 

has  been  identified) .  If  the  debugging  is  performed  at  time  t  , 
then  the  data  consists  of  the  number  of  detected  errors  due  to  the 
ith  bug  to  be  discovered. 
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(iii)  If  one  is  not  willing,  to  take  the  above  "Bayesian  approach",  one 
could  try  maximum  likelihood  estimates.  For  instance,  if  all 
debugging  is  performed  at  time  t  ,  then  with  N^(t)  denoting 
the  number  of  detected  errors  caused  by  bug  i  ,  the  estimator 


-N  (t) 

.  N  (t)e  . 

E3(t>  *  ft  1 ^  l  V‘> 

1  -  e 


is  suggested.  Whereas  additional  numerical  work  is  needed  to  see 

M^t) 

how  this  estimator  compares  with  — — —  ,  preliminary  simulation 
investigations  show  that  it  compares  quite  favorably  (see  Table  1). 

(iv)  Suppose  that  the  nature  of  the  problem  is  such  that  a  debugging 

must  take  place  whenever  an  error  is  detected  and  the  bug  removed. 
If  we  are  able  to  determine  the  bug's  failure  rate,  then  this 
case  reduces  to  one  originally  considered  in  this  section. 

However,  let  us  now  suppose  that  the  bugs  discovery  sheds  no 
light  on  its  failure  rate.  If  ,  i  =  1,  ...,  R  denote  the 
times  at  which  the  detected  errors  occur,  then  the  MLE  of 
A^,  ...»  Ar  is  1/T^,  ...,  1/T^  .  Hence  a  natural  estimator 
to  consider  in  this  case  is 


where  =  T^/t  . 


(v)  The  following  is  a  partial  summary  of  extensive  simulations. 


TABLE  1 

Summary  of  Simulation  Results:  Average  values  based  on  100  simulations:  p=t 

"Ai 

R  A  e 

E1  "  M1  •  E2  =  l  ~  -A~ 

i*l  ,  i 

1  -  e 


.  -j  R  -1/T 

E,  -  l  Mj  .  e4  -  l  «  1 

J  1  -  e  J  i-1 


Tl(i  -  e 


Average  values:  based  on  100  simulations 
|  (E,-A(l))2  I  (E_-A(l))2  I  (E.-A (1) )2  I  (E.-A(l)) 


50  bugs 

unif 

(0,3) 

Case 

1 

Case 

2 

50  bugs 

unif 

(0,4) 

Case 

1 

Case 

2 

50  bugs 

unif 

(0,5) 

Case 

1 

Case 

2 

Case 

3 

Case 

4 

Case 

5 

Case 

6 

50  bugs 

unif 

(0,6) 

Case 

1 

Case 

2 

Case 

3 

Case 

4 

50  bugs 

unif 

(0,7) 

Case 

1 

Case 

2 

35.364 

38.678 

30.311 

36.569 

19.881 

30.431 

36.133 

29.588 

30.693 

23.772 

21.528 

21.284 

25.738 

21.306 

33.928 

28.415 


23. 

.753 

27. 

.279 

23. 

,772 

in 

CM 

.814 

15. 

,914 

27. 

,723 

25. 

,341 

22, 

,255 

25, 

,802 

18. 

,534 

15. 

,989 

18. 

,854 

21. 

,613 

20. 

,769 

25, 

,449 

21. 

,366 

26.155 

27.559 

25.027 

26.169 

16.593 

27.061 

26.418 

24.368 

27.712 

21.588 

17.998 

17.462 

21.447 

18.973 

26.865 

22.286 


60.282 

68.528 

53.324 

64.773 

27.751 

40.583 

48.156 

25.996 

34.181 

22.915 

18.658 

29.050 

30.873 

24.786 

34.641 

29.503 


(vi)  The  results  of  this  section  are  consistent  with  the  hypothesis 
that  in  attempting  to  estimate  A(t)  the  best  one  can  do  is 
to  estimate  E [ A (t) ]  .  That  is,  suppose  one  somehow  knows  the 
r  “XiPt 

value  l  A  e  .  Then  the  author  suggests  that  the  data  should 

i  1 

-  "Xipt 

be  ignored  and  A(t)  should  be  estimated  by  l  A  e  .  As 


Var  (A(t)) 


-xitp) 


2  ‘VP 


-x1tp 


1  -  e 


it  follows  that  E[A(t)]  is  a  better  estimate  of  A(t)  than  is 


-A±tp 


R  A±e 

£  - t~  .  Of  course  E  [ A  (t)  ]  is  unknown  and  cannot  be 

1  -  e  1 

directly  implemented  as  an  estimator.  (The  results  of  Section  4 
also  indicates  that,  in  effect,  we  are  really  trying  to  estimate 
E [ A (t) ]  and  not  A(t)  . 


ALLOWING  FOR  UNSUCCESSFUL  DEBUGGINGS 


Suppose  as  in  the  previous  section,  that  detected  errors  lead  to  im¬ 
mediate  debugging  with  the  failure  rate  of  the  responsible  bug  being  deter¬ 
mined.  However,  let  us  now  suppose  that  the  debugging  is  only  successful 
with  probability  a  .  That  is,  with  probability  1  -  a  a  new  bug,  which 
we  will  suppose  has  the  same  failure  rate  as  the  bug  just  removed,  is 
created.  (Thus  we  can  think  of  the  newly  created  bug  as  either  being  the 
old  bug  which  was  not  successfully  eliminated  or  as  being  a  brand  new  bug 
caused  by  our  change  in  the  program  that  eliminated  the  old  bug  and  which 
has  the  same  failure  rate  as  the  old  bug) .  Suppose  also  that  when  a 
debugging  takes  place,  we  are  able  to  tell  whether  the  responsible  bug  was 
initially  present  or  was  created  by  a  previous  debugging. 

We  can  estimate  the  failure  rate  at  time  t  as  follows:  Let  us  start 
by  adopting  the  interpretation  that  if  a  debugging  is  unsuccessful,  then  the 
responsible  bug  remains  in  the  program.  That  is,  we  are  identifying  any 
newly  created  bug  as  being  identical  to  the  one  removed.  Suppose  that  R 
distinct  bugs — having  failure  rates  A^,  ...»  AR — have  been  discovered,  with 

L^  ,  i  =  1,  ....  R  representing  the  last  time  that  the  bug  with  rate 

-A  tp 
R  A^e  1 

has  been  responsible  for  a  detected  error.  Then  )  - — —  estimates 

i«i  ,  'V 

the  error  rate  at  time  t  of  those  bugs  that  have  not  yet  appeared.  Also 


the  discovered  bug  with  rate  A^  will  still  be  present  in  the  package  at 


time  t  with  probability 


-A , (t-L . ) 
e  1  1  (1  -  q) 


a  +  (1  -  a)e 


-Ai(t-Li) 


.  Hence  the  total  rate  at 


time  t  can  be  estimated  from 
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4.  A  BAYESIAN  MODEL 

To  formulate  a  Bayesian  model,  we  need  specify  a  prior  joint  distribu¬ 
tion  for  m  ,  X^,  . . . ,  X^  .  As  there  are  a  large  number  of  possible  things 

that  could  go  wrong  when  putting  together  a  software  package,  each  having  a 
small  probability  of  going  wrong,  it  seems  reasonable  to  suppose  that  m  , 
the  number  of  bugs,  has  a  Poisson  distribution.  Also,  given  m  ,  we  shall 
suppose  that  the  resulting  failure  rates  are  independent  and  identically 
distributed.  So  let  us  make  the  following  assumption. 


Assumption: 

The  number  of  bugs  m  has  a  Poisson  distribution  with  mean  c  ;  and 
given  m  ,  X^,  ...,  Xffi  are  independent  and  have  the  common  distribution 

G  .  Both  c  and  G  are  assumed  to  be  known. 

We  shall  assume  that  once  a  bug  is  detected,  its  failure  rate  becomes 
known  and  the  bug  is  eliminated.  (That  is,  a  of  Section  3  is  taken  to 
equal  1) . 

As  each  of  the  Poisson  number  of  bugs  will  independently  result  in  a 
detected  error  with  probability  given  by 

P{bug  has  a  detected  error}  *  J(  1  -  e"Xpt)dG(X) 

it  follows  that  the  number  of  discovered  bugs  is  Poisson  with  mean 
c  J  (1  -  e  ^pt)dG(X)  and  is  independent  of  the  number  of  undetected  bugs 
which  is  Poisson  with  mean  c  J  e  ^ptdG(X)  .  Also  the  conditional  distribu¬ 
tion  of  a  bugs  failure  rate,  given  that  the  bug  is  not  discovered  is  as 
follows: 


dG(X  |  not  discovered) 


JV>ptdG(X>  ' 
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Hence, 


E[A  |  not  discovered]  = 


Ae~XptdG(A) 

VXptdG(A) 


and  thus 


E[A(t)  I  data]  =  c  I  e  pcdG(A)E[A  I  not  discovered] 


c  f Ae"XptdG(A)  . 


That  is,  the  Bayes  estimator  with  respect  to  square  error  loss  is  independent 
of  the  data  and  is  as  given  above. 


4 . 1  Unknown  c 


If  c  is  unknown  in  the  above  model,  then  we  can  estimate  it  by 


;  a- 


e"  P  )dG(A) 


where  R  is  the  number  of  discovered  bugs  |and  thus 


has  a  Poisson  distribution  with  mean  c  /  (1  -  e  Xpt)dG(A)|.  Hence,  we  can 


estimate  the  Bayes  estimator  as  follows: 


E[A(t)  I  data]  e=t 


R  J  Ae"XptdG(A) 
f (1  -  e"Xpt)dG(A) 


4.2  Both  c  And  G  Unknown 


Note  first  that  the  conditional  distribution  of  the  fault  rate  of  a 


bug  that  is  discovered  is  as  follows: 


dG(A  discovered) 


(1  -  e~Xpt)dG(A)  _ 
f (1  -  e"Xpt)dG(A)  ’ 


y. -\y.ylyl -\y«y.y.y.y 


5.  ESTIMATING  A  FUTURE  FAILURE  RATE 


Using  the  same  notation  as  in  Section  4,  suppose  we  are  now  interested 
in  estimating  at  time  t  what  the  failure  rate  would  be  at  time  t  +  s  if 
testing  is  continued  for  an  additional  s  time  units.  As  in  Section  4, 
the  Bayes  estimate  of  A(t  +  s)  is,  when  c  and  G  are  known. 


E[A(t  +  s)  |  data  to  t]  =  c  |  Xe  Xp(t+s)dG(X) 


J 


Now  suppose  c  and  G  are  unknown.  VJe  can  estimate  c  as  before  from 


est 


J  (1  -  e'Up)dG(A) 


where  R  is  the  number  of  bugs  discovered  by  t  .  Also  as 


Ae-Ap(t+s) 


1  -  e 


-Apt 


discovered  by  t  = 


JAe-lp(t+s)dG(X) 

fa  -  e'ipt)dG(») 


we  see  that 


(3) 


R  A  .e 

E[A(t  +  s)  |  data  to  t]  e=t  £  - 

i=l 


-Aip(t+s) 


1  -  e 


-A-jPt 


where  A^,  ...»  AR  are  the  discovered  failure  rates  by  t  . 

If  we  now  forget  the  Bayesian  scenario  that  led  to  the  above  estimator 
and  consider  it  from  a  more  classical  approach,  we  obtain  that 
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-A  p(t+s) 

R  A.e 


-Aip(t+s) 

=  e  l  — — "  *i<t)) 

i  i  i 

1  -  e 

-A  p(t+s) 

=  l  A  e  =  E[A(t  +  s)] 

i 


Also, 


(!  -A  p(t+s)  ^ 

R  A  e 

k  "  Mt  +  s) 

1  -  e  i 


(<  -A±p(t+s)  ^ 

I  ~ — r  ^i(t)  +  l  Wc  +  s) 

1  -  e  ) 


-2A  p(t+s) 

A  .  -A  pt  ->■  Pitts),  -A  p(t+s) ^ 

'  l - ^T-e  +pie  I1'6  1 

1  1  -  e 


?  -2A  p(t+s) 

+  2lAe  1 
i 

,  -a  p(t+s)  r  -xi',<t+s) 

i iie  1 + — ^ 

1  -  e 


The  above  estimate  presupposes  that  a  bugs  failure  rate  becomes  known 
when  the  bug  is  discovered.  If  this  is  not  the  case,  then  we  can  still  use 
the  data  obtained  by  t  to  estimate  A(t  +  s)  .  One  approach  is  to  note 


N.(t) 

* 


where  N^(t)  is  the  number  of  detected  errors  caused  by  bug  i  .  Hence, 
using  (3)  we  can  estimate  A(t  +  s)  by 
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-Ni(t)p(t+s)/t 


.  est  1  ,  Vt)e 

(t  ’  "  t[  -N^Op 

1  -  e 


"T  I  V0 

C  i=l  1 


-ip(l+s/t) 


1  -  e 


A  second  approach  to  estimating  A(t  +  s)  is  to  note  the  following: 

-X  p(t+s) 

E[A(t  +  s)]  =  l  X±e 


r  .  'Xi PC/,  ,  ,  <XiPS)  <XiPS)  ,  \ 

=  2,  *±e  ll  -  xiPs  +  — 2 - 3i —  +  •  •  ■) 

1  ”  M  <t)  i  1  i+1’ 

-  -  e  y  is1 1(-Di+1  . 

p  i=l  t1 


The  last  equality  following  since 


E[M  <t)J  =  l 
J  i 


-A±pt  (X^t)^ 


Hence  the  above  suggests  the  possible  estimator 


,  00  M.  ( t)  .  -  ... 

, x  est  1  v  i  ,  i-1, 

A(t  +  s)  =  -  I  - —  is  (-1) 

P  i=l  t 


Though  we  intuitively  favor  (4),  numerical  tests  are  needed  to  see  whether 
(4)  or  (5)  yields  the  better  estimates.  Of  course,  s  should  not  be  too 
large  in  relation  to  t  for  either  estimate  to  be  very  effective. 

We  can  also  use  the  above  to  estimate  the  expected  number  of  new  bugs 


discovered  in  (t  ,  t  +  s)  .  As  this  quantity  is  equal  to  pE^J  A(t  +  y)dy|  , 
it  follows  from  (3)  that  we  can  estimate  this  quantity  by 


.  •  ■  •  V  V 


WWW? 


*•  „**»  •**  - 


When  the  failure  rates  do  not  become  known  when  a  bug  is  detected,  we  can 
estimate  the  expected  number  of  bugs  that  will  be  discovered  between  t 
and  t  +  s  by 

l  M.(t>  — ^■1P,-n  (1  -  e'll,s/t)  . 
i  1  1  -  e"ip 

Remarks : 

(i)  The  results  in  this  section  can  be  used  to  devise  an  easily  imple¬ 
mented  stopping  rule  for  testing.  One  could  test  for  a  time  t  and 
then  based  on  the  observed  data  choose  an  additional  time  testing  time 
s  such  that  the  estimated  error  rate  at  s  would  be  acceptable.  One 
can  then  reevaluate  this  after  testing  for  the  additional  time  to  de¬ 
termine  whether  to  stop  or  to  continue  for  an  additional  time  indi¬ 
cated  by  the  above. 
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